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Abstract 

The paper presents a pilot study conducted with spatial visual, audiovisual and auditory brain-computer- 
interfacc (BCI) based speller paradigms. The psychophysical experiments arc conducted with healthy 
subjects in order to evaluate a difficulty and a possible response accuracy variability. We also present 
preliminary EEG results in offline BCI mode. The obtained results validate a thesis, that spatial auditory 
only paradigm performs as good as the traditional visual and audiovisual speller BCI tasks. 

1 Introduction 

The brain computer and brain machine interfaces (BCI and BMI) pQ, or generally the brain evoked re- 
sponses to any external stimuli are usually based on the monitoring of the neuro- and electro-physiological 
activity by means of the electroencephalogram (EEG) |2. Due the method's non-invasive nature, the 
EEG based BCI paradigms are perfectly suited to be at the core of future "intelligent" interfacings and 
neuro-prosthetic developments. The concepts are particularly fitted to the needs of the handicapped as 
well the cores of the smart environments, computer gaming and virtual reality applications. 

A concept of utilizing brain various sensory modalities creates a very interesting possibility to let the 
subjects to interact with more rich multi-sensory environments, which possibly shall result with better 
brain-wave-responses classification and information-transfer-rates (ITR). 

The auditory BCI (aBCI) is thus potentially a less mentally demanding paradigm receiving recently 
more attention in auditory neuroscience applications [SHZ]- We propose to compare first psychophysical 
responses to various audio or visual modality reposes in application to a spatial perception typer/speller 
paradigm. The results shall next help to design a more comfortable BCI spellcr/typer paradigm. 
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Next wc present the preliminary results from EEG experiments where the evoked responses to the 
various modality in form of event related potentials (ERP) are compared for target and non-target 
stimuli. 

In the following sections we discuss audio, visual and audiovisual spatial speller BCI paradigm interface 
applications. Next we describe the psychophysical and EEG offline BCI experiments conducted for 
auditory, visual and audiovisual cases. A statistical analysis of the obtained results and a discussion 
conclude the paper. 

2 Methods 

The experiments are conducted in a combination of BCI2000 environment [5] to generate stimuli patterns 
and Max/MSP by Cycling'74 software to analyze subjects behavioral responses in form of button presses 
to instructed target letter directions in visual, auditory and audiovisual modes. For the psychophysical 
and the later EEG BCI experiments the stimuli in various modalities are generated as follows. 

2.1 The psychophysical experiments 

The psychophysical experiment series is conducted with three subjects seating in front of computer display 
where instructions and visual stimuli are given. The subjects respond by pressing a button immediately 
after each target's appearance. The three tested modality setting details are as follows. 

2.1.1 The visual speller psychophysical task 

Five Japanese hiragana letters - a, i, u, e, o - are flashed in a random order. The subject are instructed 
to attend (press a response button) to the target letter presented in each random trial sequence. The 
non-target stimuli shall be ignored. 

2.1.2 The auditory speller psychophysical task 

The same Japanese hiragana letters are delivered as synthetically generated sounds from two loudspeakers 
positioned in front of the subjects at a one meter distance and —45° and 45° azimuths. A vector based 
amplitude panning method (VBAP) [S] is used to virtually distribute sound images at the positions of 
—90°, —45°, 0°, 45°, 90°. The sound levels are uniform for all the letters at 70 dB. Here again subjects are 
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instructed to attend (press the response button) as soon as they could hear a target letter in a randomly 
presented series. 

2.1.3 The audiovisual speller psychophysical task 

The audiovisual task involves presentation of the both above described auditory and visual stimuli letters 
simultaneously creating a multimodal paradigm. The instructions given to the subjects are the same as 
in the both unimodal tasks. 

2.2 The EEG experiment with offline BCI protocol 

We conduct experiments to record EEG for further offline analysis of the brain responses to visual, 
auditory and audiovisual modalities. The purpose is to find how much the P300 EEG potential (a 
so called "aha" response) variability, for target versus non-target discrimination in each modality, is 
modulated by various sensory stimuli in the same spatial speller task. The EEG signals are captured 
with eight dry electrodes portable wireless system by g.tec (g. SAHARA &l g.MOBIIab+). The recorded 
EEG at each electrode is preprocessed by BCI2000 application. The sampling rate is 256 Hz and the 
notch filter to remove electric power lines interface of 50 Hz is set in a band of 48 — 52 Hz, according to 
the East Japan power stations specification. The EEG experiments are conducted with agreement of the 
institutional ethical committee guidelines for experiments with human subjects. The EEG experimental 
protocol is as follows: 

2.2.1 The visual speller offline BCI task 

Five Japanese hiragana letters are flashed in a random order. The subjects are instructed to attend and 
count the target letters presented in each random trial, while ignoring the non-targets. Since only the 
EEG evoked response are of the experimental interest, the subjects are requested not to move and to 
limit eye blinking. 

2.2.2 The auditory speller offline BCI task 

The same Japanese hiragana letters are delivered as synthetic generated sounds from two loudspeakers 
positioned in front of the subjects at a one meter distance at the —45° and 45° azimuths. This experiment 
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is conducted with the conditions as the above visual stimuh EEG recording, thus the subjects receive the 
same instructions. 

2.2.3 The audiovisual speller offline BCI task 

The audiovisual task involves presentation of the both auditory and visual stimuli letters as in the previous 
sections. The subject is given the same instruction to attend and count the audiovisual presented target 
letter as in above unmoral offline BCI tasks. 

3 Results 

The results of the psychophysical experiments in the three modality spatial speller settings have shown 
no significant differences among median response delays. The visual, auditory and audiovisual modalities 
the have similar cognitive loads resulting with the same behavioral markers. There has been no significant 
median response differences in pairwise t- and Wilcoxon-tests. The grand mean results for all subjects have 
been presented in Figures [5]-[5]for auditory, visual and audiovisual modalities respectively. A summary plot 
comparing all three modalities with median values for all stimuli has been presented in Figure[5] A median 
value of mean psychophysical responses, in the above figure, for auditory modality lies between audiovisual 
and visual modalities. Here again the differences among median values has been not significant. 

The collected EEG signals have been analyzed offline using P300GUI [TU] classification toolbox. The 
results of the analysis has been presented in Figures [BHHl The top panels in each of the offline BCI EEG 
experiment results figure present the scalp topographies for targets at an ERP latency related to the 
maximum difference between target and non-target responses. The middle panels of the result figures 
visualize the differences between mean target and non-target responses as the time series. The visual and 
audiovisual modalities (Figures [7] and HI respectively) have resulted with the classical P300 responses with 
positive deflection after 300 ms from the stimulus onset. The auditory only modality resulted with a very 
interesting earlier separable component (see Figure [S]) starting around 250 ms after the stimulus onset, 
which has an opposite polarity comparing the the vision related modalities. This is a very interesting 
result for the future development of the auditory modality only based BCI paradigms. 
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4 Conclusions 

The results of the conducted psychophysical experiments have shown that the auditory modality only has 
the same cognitive load in the BCI paradigm application as the traditional visual or audiovisual cases. 
The good news is that the less explored, in BCI research, auditory spatial modality could create a perfect 
alternative for the subjects with limited or completely lost vision, which situation is common in case of 
totally-locked-in~syndrome patients. 

The presented preliminary EEG responses in the same offline BCI setting confirmed also the hypothesis 
of the possible successful replacement of the traditional spatial visual or audiovisual paradigms with the 
auditory one. We also presented the very interesting and new early auditory response, which could 
possibly speed up and enhance the intentional response classification in auditory modality only BCI 
applications. 

The presented research and results are a step forward in the dynamic state-of-the-art research leading 
into development of less mentally demanding and with higher accuracy BCI paradigms which are very 
much awaited by the patients in need. 

We plan the following series of experiments with larger number of subjects to further test and optimize 
the proposed spatial auditory speller BCI paradigm, which will also include a larger number of Japanese 
letters to be spelled. 
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Figure Legends 

Figure [l] Visual, auditory and audiovisual BCI speller test paradigm environment based on BCI2000 soft- 
ware [8j. The subject in the picture wears also EEG g. SAHARA electrodes connected to g.MOBIIab+ 
amplifier by g.tec. In this paper we conduct the psychophysical and the preliminary EEG, in offline 
BCI setting, experiments to evaluate possible differences related to various stimulus perceptions. 

Figure [2] Auditory modality only psychophysical experiment results in form of boxplots with response 
distributions of the five different stimuli. The differences in response times among stimuli directions 
are not significant. 

Figure [3] Visual modality only psychophysical experiment results in form of boxplots with response 
distributions of the five different stimuli. The differences in response times among stimuli directions 
are not significant. 

Figure |4] Audiovisual modality psychophysical experiment results in form of boxplots with response 
distributions of the five different stimuli. The differences in response times among stimuli directions 
are not significant. 

Figure [5] Grand mean for all modality psychophysical experiments as in Figures [UHl The median 
response time delays to the proposed auditory only modality are between audiovisual (the fastest) 
and visual (the slowest) results. 

Figure [6] Auditory modality only offline BCI EEG responses presented as scalp mean topography for 
targets - in the top panel; as the mean time series with standard deviation error-bars for non-targets 
(green) and targets (red) in the middle panel; and as the signed statistical difference (i.e., signed 
r^) value which evaluates the discriminability between the two types of ERPs - in the bottom panel. 
The figure was created with P300GUI [TU] . 

Figure [7] Visual modality only offline BCI EEG responses presented as scalp mean topography for targets 
- in the top panel; as the mean time series with standard deviation error-bars for non-targets (green) 
and targets (red) - in the middle panel; and as the signed statistical difference (i.e., signed r^) value 
which evaluates the discriminability between the two types of ERPs - in the bottom panel. The 
figure was created with P300GUI [10] . 
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Figure [8] Audiovisual modality only offline BCI EEG responses presented as scalp mean topography for 
targets - in the top panel; as the mean time series with standard deviation error-bars for non-targets 
(green) and targets (red) in the middle panel; and as the signed statistical difference (i.e., signed 
r^) value which evaluates the discriminability between the two types of ERPs - in the bottom panel. 
The figure was created with P300GUI [10]. 
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Figure 1. Visual, auditory and audiovisual BCI speller test paradigm environment based on BCI2000 
software [8]. The subject in the picture wears also EEG g. SAHARA electrodes connected to g.MOBIIab+ 
amplifier by g.tec. In this paper we conduct the psychophysical and the preliminary EEG, in offline BCI 
setting, experiments to evaluate possible differences related to various stimulus perceptions. 
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Psychophysical responses to auditory only stimuli 
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Figure 2. Auditory modality only psychophysical experiment results in form of boxplots with response 
distributions of the five different stimuli. The differences in response times among stimuli directions are 
not significant. 



Psychophysical responses to visual only stimuli 
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Figure 3. Visual modality only psychophysical experiment results in form of boxplots with response 
distributions of the five different stimuli. The differences in response times among stimuli directions are 
not significant. 
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Psychophysical responses to audiovisual stimuli 
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Figure 4. Audiovisual modality psychophysical experiment results in form of boxplots with response 
distributions of the five different stimuli. The differences in response times among stimuli directions are 
not significant. 



Grand mean for the three modalities 
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Figure 5. Grand mean for all modality psychophysical experiments as in Figures [2H11 The median 
response time delays to the proposed auditory only modality are between audiovisual (the fastest) and 
visual (the slowest) results. 
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Figure 6. Auditory modality only offline BCI EEG responses presented as scalp mean topography for 
targets - in the top panel; as the mean time series with standard deviation error-bars for non-targets 
(green) and targets (red) in the middle panel; and as the signed statistical difference (i.e., signed r^) 
value which evaluates the discriminability between the two types of ERPs - in the bottom panel. The 
figure was created with P300GUI [TO] . 
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Figure 7. Visual modality only offline BCI EEG responses presented as scalp mean topography for 
targets - in the top panel; as the mean time series with standard deviation error-bars for non-targets 
(green) and targets (red) - in the middle panel; and as the signed statistical difference (i.e., signed r^) 
value which evaluates the discriminability between the two types of ERPs - in the bottom panel. The 
figure was created with P300GUI [TO] . 
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Figure 8. Audiovisual modality only offline BCI EEG responses presented as scalp mean topography 
for targets - in the top panel; as the mean time series with standard deviation error-bars for non-targets 
(green) and targets (red) in the middle panel; and as the signed statistical difference (i.e., signed r^) 
value which evaluates the discriminability between the two types of ERPs - in the bottom panel. The 
figure was created with P300GUI [TO] . 



